feat(postgresql): dynamic schema #23401

betodealmeida · 2023-03-16T22:09:26Z

SUMMARY

Builds on #23385. Renames the adjust_database_uri method to adjust_engine_params and makes it accept connect_args as well, so we can implement dynamic schema in Postgres. In addition, the method also takes an optional catalog, so in the future we can implement proper catalog support.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Before, when using Postgres the SQL Lab schema dropdown is not used to run the query. Note that in the example below information_schema is selected, but the query actually runs in the public schema:

With this PR:

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

codecov · 2023-03-16T22:23:23Z

Codecov Report

Merging #23401 (f465853) into master (42e8d1b) will increase coverage by 1.64%.
The diff coverage is 80.00%.

❗ Current head f465853 differs from pull request most recent head 2db69f2. Consider uploading reports for the commit 2db69f2 to get more accurate results

@@            Coverage Diff             @@
##           master   #23401      +/-   ##
==========================================
+ Coverage   65.96%   67.61%   +1.64%     
==========================================
  Files        1907     1907              
  Lines       73590    73611      +21     
  Branches     7982     7982              
==========================================
+ Hits        48546    49772    +1226     
+ Misses      22996    21791    -1205     
  Partials     2048     2048

Flag	Coverage Δ
hive	`52.73% <36.92%> (-0.02%)`	⬇️
mysql	`78.40% <67.69%> (+<0.01%)`	⬆️
postgres	`78.46% <67.69%> (-0.04%)`	⬇️
presto	`52.65% <38.46%> (-0.02%)`	⬇️
python	`82.32% <80.00%> (+3.39%)`	⬆️
sqlite	`76.97% <67.69%> (?)`
unit	`52.51% <58.46%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/db_engine_specs/snowflake.py	`66.86% <14.28%> (+19.27%)`	⬆️
superset/db_engine_specs/drill.py	`88.88% <40.00%> (+33.33%)`	⬆️
superset/dashboards/commands/importers/v1/utils.py	`85.00% <50.00%> (+6.79%)`	⬆️
superset/models/core.py	`89.74% <77.77%> (-0.36%)`	⬇️
superset/dashboards/schemas.py	`98.98% <83.33%> (-0.50%)`	⬇️
superset/db_engine_specs/base.py	`90.80% <100.00%> (+8.75%)`	⬆️
superset/db_engine_specs/hive.py	`87.79% <100.00%> (+1.01%)`	⬆️
superset/db_engine_specs/mysql.py	`98.79% <100.00%> (+9.63%)`	⬆️
superset/db_engine_specs/postgres.py	`97.01% <100.00%> (+5.95%)`	⬆️
superset/db_engine_specs/presto.py	`88.05% <100.00%> (+1.25%)`	⬆️

... and 81 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

eschutho · 2023-03-17T23:17:51Z

superset/db_engine_specs/base.py

@@ -1057,30 +1057,33 @@ def extract_errors(
        ]

    @classmethod
-    def adjust_database_uri(  # pylint: disable=unused-argument
+    def adjust_engine_params(  # pylint: disable=unused-argument


@betodealmeida what do you think will be the impact of this change on any custom db engine specs that have been added to Superset instances?

oh, nm, I see how you covered this below.

eschutho · 2023-03-17T23:32:32Z

superset/db_engine_specs/base.py

-    ) -> URL:
+        connect_args: Dict[str, Any],
+        catalog: Optional[str],
+        schema: Optional[str],


I noticed that in some of the class overrides, catalog and schema have kwargs with defaults instead of positional arguments. Do we want to be consistent?

Ah, good point, let me add defaults here.

eschutho

Looks great! I just had the one question about args vs kwargs, but not a blocker.

pull-request-size bot added the size/L label Mar 16, 2023

betodealmeida force-pushed the postgres_set_schema branch 2 times, most recently from 0ba2db6 to 05b1e7e Compare March 16, 2023 22:15

betodealmeida requested a review from villebro March 16, 2023 22:33

feat(postgresql): dynamic schema

c2b2644

betodealmeida changed the base branch from default_schema to master March 17, 2023 15:39

pull-request-size bot added size/XL and removed size/L labels Mar 17, 2023

betodealmeida force-pushed the postgres_set_schema branch from 05b1e7e to c807664 Compare March 17, 2023 15:40

pull-request-size bot added size/L and removed size/XL labels Mar 17, 2023

Add catalog

c4c7820

betodealmeida force-pushed the postgres_set_schema branch from c807664 to c4c7820 Compare March 17, 2023 15:42

DRY options parsing

0835da1

betodealmeida force-pushed the postgres_set_schema branch from 9a3b2f7 to 0835da1 Compare March 17, 2023 16:43

betodealmeida mentioned this pull request Mar 17, 2023

[SIP-95] Proposal for Catalog Support in SQL Lab #22862

Closed

eschutho reviewed Mar 17, 2023

View reviewed changes

eschutho approved these changes Mar 17, 2023

View reviewed changes

Add default values to schema/catalog

2db69f2

betodealmeida merged commit 2c6f581 into master Mar 18, 2023

michael-s-molina mentioned this pull request Jul 18, 2023

RDS Proxy Connection Error with Command-Line Options in SQL Lab #24684

Closed

3 tasks

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 3.0.0 labels Mar 13, 2024

mistercrunch deleted the postgres_set_schema branch March 26, 2024 16:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(postgresql): dynamic schema #23401

feat(postgresql): dynamic schema #23401

betodealmeida commented Mar 16, 2023 •

edited

Loading

codecov bot commented Mar 16, 2023 •

edited

Loading

eschutho Mar 17, 2023

eschutho Mar 17, 2023

eschutho Mar 17, 2023 •

edited

Loading

betodealmeida Mar 17, 2023

eschutho left a comment

feat(postgresql): dynamic schema #23401

feat(postgresql): dynamic schema #23401

Conversation

betodealmeida commented Mar 16, 2023 • edited Loading

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

codecov bot commented Mar 16, 2023 • edited Loading

Codecov Report

eschutho Mar 17, 2023

Choose a reason for hiding this comment

eschutho Mar 17, 2023

Choose a reason for hiding this comment

eschutho Mar 17, 2023 • edited Loading

Choose a reason for hiding this comment

betodealmeida Mar 17, 2023

Choose a reason for hiding this comment

eschutho left a comment

Choose a reason for hiding this comment

betodealmeida commented Mar 16, 2023 •

edited

Loading

codecov bot commented Mar 16, 2023 •

edited

Loading

eschutho Mar 17, 2023 •

edited

Loading